Realtime User Attention and Emotion Estimation on a Mobile Robot

Within the scope of service robotics a natural and adaptive dialog is getting more and more important to offer intuitive interaction for users not familiar with the system. Therefore, service robots have to be aware of the person’s mood and visual attention. Furthermore, robotic systems should be able to recognize simple head gestures like head shaking or nodding as these gestures are a natural way of human communication. This paper presents a method to extract these information from image sequences containing the head of the interaction partner with the help of Active Appearance Models (AAMs). Therefore, a variant of AAM robust to illumination is fitted to a sequence of face images to get a parametric description of the user’s face. The paper shows how to apply these parameters for human state estimation in terms of attention and emotion positivity. Furthermore, we show how to utilize Temporal Event Mapping with the AAM parameters to learn head gestures and head gesture semantics online during the human-robot dialog.